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ABSTRACT 

According to Keller's Model of Motivation, one 
technique for increasing learners' motivation, and thus their mental 
effort, involves increasing the personal relevance of the material. 
When instruction is perceived as relevant, learners perceive that 
important personal needs are being met by the learning situation. The 
purpose of this study was to manipulate the degree of relevance 
present in both a videotaped and print version of a lesson to 
determine whether increasing the relevance of a lesson increases the 
amount of material effort invested. Students enrolled in a sophomore 
level education course at a large Midwestern university were invited 
to participate in a two-hour lesson on desktop publishing. 
Instructional materials consisted of four lessons: (1) a "standard" 
video on desktop publishing; (2) a motivationally enhanced videotape; 
(3) a text version of the standard lesson: and (4) a te:.t version of 
the motivationally enhanced lesson. Assessment measures consisted of 
a self-report questionnaire, a computer program that measured 
response rates, and a posttest. Findings were consistent with the 
results of previous studies conducted with elementary and middle 
school students. Participants reported in the questionnaire that they 
invested more mental effort in learning from print versions of the 
lessons than in video-based versions. The students who received the 
video-based lesson had significantly longer reaction times to the 
secondary task measure. (Contains 26 references.) (AEF) 
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The Effects of Relevan ce on Mental Effort 

It is 3 corninonly accepted that learners view video as an easy, passive tnedium, and thus invest little mental 
effort in processing the information (the "couch potato"). This assumption is primarily based on a series of studies that 
Salomon (Salomon, 1983; Salomon, 1984; Salomon & Leigh, 1984) conducted in the early eighties. In diese studies, 
Salomon found that learners perceived it to be "easier" to learn from television than books, reported spending less mental 
effort in learning from videotape than from books, and recalled less from a video-based version of a story than from a 
print -based version. Salomon assumed that learners perceptions of video or television as "easy" resulted in the 
investment of less mental effort, and consequently less learning achievement, when compared to reding books. 

We could assume from these conclusions that it is preferable to provide instruction in a printed form; however, 
video remains a popular means of conveying instruction, particularly in the training environment. In fact, a recent 
survey conducted by Training magazine reported that the use of videotape for conducting corporate training was exceeded 

only by stand-up instruction (Froiland, 1993). 

The majority of research investigating the mental effort iii vested in instructional videotapes has been conducted 
with elementary and middle school children (Beentjes & van deer Voort, 1993; Sherman, Salomon, 1983; Salomon, 

1984; Salomon & Leigh, 1984). Although several researchers have investigated the amount of effort that college 
students invest in processing print lessons (Britton, Muth, & Glynn, 1986; Britton, Piha, Davis, & Wehausen, 1978, 
and others) and television commercials (Lang, Geiger, Strickwerda, & Sumner, 1993; Reeves & Thorson, 1986; and 
others), there has been very little research that has systematically investigated the mental effort older learners expend in 
learning from video-based instructional materials. As older learners may have more well developed strategies for learning 
from videotape that elementary and middle school students, it is important to investigate the extent to which Salomon's 

conclusions £q>ply to the adult population. . , . i- j 

Salomon (1983a) defines the f'onstruct of "mental effort" as the "number of non-automatic elaborations applied 
to a unit of material" (p. 42). In contrast to automatic processing that is fast and effortless, non-automatic processing is 
deliberate, conscious, and very much under the control of the individual. As new information is received, the new 
information cues the retrieval of related prior knowledge (E. Gagne, 1985) stored in the learner's schemata. The learner is 
then able to make connections between the new information and information retrieved from prior knowledge; this ptwess 
is referred to as elaboration (E. Gagne, 1985). The increased contact with the learner's mental schemata that results from 
the conscious, non-automatic generation of elaborations, or mental effort, is presumed to facilitate the retention and 

retrieval of the new material. . u- j ^ 

The search for techniques to increase the effort that learners invest in video-based instruction has been hindered 

by the limitations of the instruments used to assess the construct of mental effort. Early work (Salomon, 1983; 

Salomon. 1984; Salomon & Leigh, 1984) that investigated the amount of effort learners invested in print and video-based 
instruction used self-report questionnaires to document the learners' effort expenditures. However recently, researchers 
(Beentjes, 1989; Cennamo, Savenye, & Smith, 1991) have identified a need for more precise methods of assessing 
mental effort. Beentjes (1989) accurately states that "... validation studies in which mental effort is assessed by 

multiple methods are called for." ( p. 56). . ^ • u o. 

Secondary task techniques have been used to assess effort in a variety studies using pnnt (Bntton, Muth, & 
Glynn 1986- Britton, Piha, Davis & Wehausen, 1978; and others) and television commercials (Lang, Geiger, 
Strickwerda,’* Sumner, 1993; Reeves & Thorson, 1986; and others), and seem to offer a promising method of assessing 
the mental effort invested in video-based lessons. When using the secondary task technique, a participant completes a 
primary task (reading, watching a video, etc.) and at the same time is instructed to respond to a secondary task such as a 
beep or flash of light as quickly as possible. Researchers assume that as learners spend more cognitive resources on the 
primary task, they have less cognitive resources left to allocate to the secondary task. They assume that slower reaction 
times to the secondary task indicate that additional cognitive effort is being allocated to the primary task. 

Although Salomon's questionnaire and the secondary task technique are assumed to both measure mental effort, 
the results of a recent study call that assumption into question. Beentjas and van der Voort (1993) assessed the effort that 
fourth and sixth grade students invested in two print-based stories and the "structurally equivalent" video-based versions. 
While students were reading or viewing the stories, they responded to a secondary task, an audible beep, by pressing a key 
as quickly as possible after hearing the beep. After they completed the stories, they responded to a self-report 
questionnaire of mental effort that was based on Salomon's instrument. Then they completed an achie/ement test. The 
researchers found very low correlaUons between students scores on the questionnaire and the secondary task measures of 
effort. In addition, the correlations between scores on both measures and achievement scores on a test of recall inferences 
were also very low. To further complicate the search for a valid means of assessing mental effort, students reported 
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spending significantly more effort in reading die text versions tlian in viewing the videotapes. However, when reaction 
times to the secondary task were used as a measure of mental effort- the opposite effects were found; reaction times were 
significantly longer, indicating more mental effort, for the videotaped versions than the text versions. Beentjas and van 
der Voort state that further research is needed on the validity of both methods of assessing mental effort. They suggest 
that one promising area of investigation may be to determine to what extent both measures are sensitive to 
manipulations that are intended to increase mental effort. 

But what manipulation? No prior studies have manipulated variables designed to increase mental effort and 
assessed increases in effort using the secondary task technique and a questionnaire. To select a manipulation that has been 
shown to increase mental effort as measured by a self-report questionnaire (fun/leam for example) may be biased toward 
demonstrating effects on the questionnaire. To select a variable that has been shown to increase mental effort as measured 
by the secondary task technique (related or unrelated cuts, for example) may be biased toward demonstrating effects on the 
secondary task measure . To further complicate the selection of variables, the vast majority of research that has used 
reaction times to a secondary task as a dependent variable has manipulated features of cither text Q£ video, not both. To 
select a variable that is effecti\ e in print may bias the results toward print; conversely, to select a variable that is effective 
in videotape may bias the results toward that medium. 

Mental effort refers to the choic e on the part of the learner to allocate cognitive resources to elaborating on an 
idea, thus, it has motivational as well as cognitive components (Salomon, 1983). According to Keller (1983), 
motivation ’’refers to the choices people make as to what experiences or goals they will approach or avoid, and the degree 
of effort they will exert in that respect." (p. 389, italics in original). He further distinguishes between effort and 
performance: ''Performance means actual accomplishment, whereas ^ort refers to whether the individual is engaged in 
actions aimed at accomplishing the task. Thus effort is a direct indicator of motivation" (p. 391). Based on the literature 
in motivational theory, Keller’s ARCS Model of Motivation (1983) breaks the concept of motivation down into four 
parts (attention, relevance, confidence and satisfaction). Keller provides specific strategies that instructional designers can 
use to increase learner’s motivation, or the effort expended in accomplishing an instructional task. 

According to Keller's (1983) ARCS model, one promising technique for increasing learner’s motivation, and 
thus their mental effort, involves increasing the personal relevance of the material. When instruction is perceived as 
relevant, learners perceive that important personal needs are being met by the learning situation" (p.406, italics in 
original) 

Nwagbara (1993) presented learners with two versions of a videotape to examine the effects of relevance on 
learners motivation toward learning the steps involved in creating a document using a desktop publishing system. Half 
the learners received the "standard" videotape and half received a ’’motivationally enhanced" version of the program that 
included scenes to increase the relevance of the lesson for the learners. He found that learners reported a greater 
willingness to expend effort in watching the motivationally enhanced videotape than in watching the same program 
without the motivational enhancements. Although the actual effort expended in the lesson was not assessed, these results 
suggest that increasing the perceived relevance of a lesson may encourage learners to invest more mental effort in learning 
the material. 

The purpose of this study was to manipulate the degree of "relevance" present in both a videotaped and print 
versions of a lesson to determine whether increasing the relevance of a lesson increases the amount of mental effort 
invested in the lesson. In addition, mental effort was assessed using both the secondary task technique and a self-report 
questionnaire to determine the more valid way of assessing mental effort. And finally, the study was conducted to 
determine the extent to which Salmon’s conclusions apply to learners of college age. The following research questions 
wm addressed: 

a) Do learners invest more effort in print-based versions of the lessons than in video-based lessons? 

b) Do learners invest more mental effort in motivationally enhanced lessons than in "standard" lessons which 
lack these characteristics? 

c) D<» learners’ self-reports of effort correlate with their "on-line" measurements of effort collected using a 
secv->ndarv task technique? 

d) Do lc*in»ers mental effort scores (from secondary task or self-report measures) correlate with their post-test 
scores on recall measures 



Methods 

For this study, a 2 (text vs. video) X 2 (enhanced vs. standard versions) X 2 (secondary task vs. non-secondary 
task) design was used. Eight sessions were scheduled over a two week period with treatments randomly assigned to each 
of the eight sessions. 
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Participants 

Students enrolled in a sophomore level education course at a large Midwestern University were invited to 
participate in a two hour lesson on desktop publishing. Of 225 students invited to participate, 130 students volunteered 
for the study. Thirteen cases were eliminated due to incomplete data (failed to complete the mental effort questionnaire, 
failure to respond to the secondary task, or to problems with the data collection equipment). After eliminating 
incomplete data sets, 1 17 participants remained in the study. Participants were both male (N= 35) and female (N=82). 

Instructiona l Materials: 

Instructional materials consisted of four lessons: a) a "standard" video on desktop publishing, b) a 
motivationally enhanced videotape , c) a text version of the "standard" lesson, and d) a text version of the motivationally 
enhanced lesson. 

The two videotapes were used in Nwagbara's (1993) previous study. The "standard" video tape is a 
commercially available product. The motivationally enhanced videoUqjc was created by editing segments designed to 
increase the personal relevance of the material into the commercial videotape. 

Keller (1987) suggests that to increase the relevance of a lesson, instructional designers should provide goal 
orientation, motive matching, and familiarity. Goal orientation can be created by relating the benefits of instruction to 
getting a job, getting a promotion, or improved job performance, or other goals that may be of value to the learners. 

The goals of the lesson should be clearly presented or learners should have the opportunity to set their own goals. In 
addit! m, learners must understand how the concepts and skills are related to their present or future goals. For learners 
who are present oriented, the instructional content can be related to their current interests. For future oriented learners, 
the instruction must provide a rationale as to how the information will be helpful in the future. Motive matching can be 
enhanced by providing personal achievement opportunities, cooperative activities, and positive role models. Learners 
should be encouraged to assume responsibility for his or her own behavior and the materials should promote perceptions 
of self-improvement on the part of the learners (McClelland, 1965; Keller, 1983). Familiarity can be increased by using 
concrete examples from settings familiar to the learner; stories, pictures, or testimonials about specific people or things 
familiar to the learner; and through the use of analogies (Keller, 1984). 

Following Keller's (1983; 1987) recommendations, motivational elements were added to a) illustrate concrete 
and provide practical examples, b) provide goals for the learners, c) provide positive role models through testimonials 
from product users, d) illustrate how the lesson might help the learner improve their present and future skills, e) 
encourage learners to assume responsibility for their learning, and e) illustrate how the skills are useful. For example, a 
section was added that stated: 

" Some of you may have used the computer at home or in school to do some assignments, write term 
papers, and other interesting things. You'll see that some of the things you already know about computers, like 
how to turn the power on, how to use the mouse, or your familiarity with the buttons on the key board will be 
handy and helpful to you in the process." (Familiarity) 

"Gaining the knowledge and skills in this presentation will open a variety of windows of opportunities for 
you in the world of work in terms of good jobs, good salaries, and of course, prospects for promotions whether 
you chose to work in the industry or as the owner of your won desktop publishing business." (goal orientation) 

In addition, Nwagbara a) removed non-instructional graphic images, b) condensed time between the narrators and 
c) eliminated or reduced the time between illustrations of the computer screens to result in two versions of the lesson that 
were both approximately 35 minutes in length. 

For this study, text versions were prepared from Nwagbara's tapes. The text of the print versions was prepared 
by transcribing the narration from the videotapes. Two undergraduate students viewed the videotapes and read the text 
versions to identify visual images that they believed to be important to the text. Based on their recommendations, 35 
images were digitized from the videotape and added to the text to provide interest and illustrate key points. For example, 
the text booklet begins with a series of images created using desktop p''’.>iishing packages. Visual images of the 
"speaker" accompany the testimonials. And finally, the text versions i iclude numerous images of the computer screen. 
The verbatim transcription was modified slightly for the print version. When the narrator makes statements such as 
"now you see" . the text version makes reference to the appropriate figure (in Figure 1, you see.."). 

5 
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Text versions were analyzed to ensure that they were equivalent to the videotaped versions. One by one, four 
graduate students in instructional design independently reviewed the text and video materials. Each student reviewed the 
regular text and video versions together, then the motivationally enhanced materials were reviewed together. As 
differences were identified between text and video versions, revisions were made to the text version. After two rounds of 
revisions, two reviewers agreed that the text and video versions were equivalent. 

Assessment measures 

5telf-report qiiestionnaire : The self-report measure of effort was created by modifying questions used by other 
researchers (Salomon, 1983b; Salomon & Leigh; 1984; Sherman, 1993). When Salomon's questions were used in a 
previous study with college students, the instrument only had Chronbach's alpha of .55 (Cennamo, Savenye, & Smith, 
1991 ). Beentjas and van der Voort ( 1993) reported alphas of .60 to .64 when using similar questions with children. 

Thus, the instrument was modified in an attempt to increase the reliability of the instrument. Four of the questions were 
modifications of questions used by Salomon (1983b, 1984) and five questions were modifications of questions used by 
Sherman (1993) who reported a Chronbach's alpha of .69 for his mental effort questionnaire. Participants were asked to 
respond to questions such as "While reading the booklet, I concentrated hard on what was shown in the pictures." "The 
lesson made me think very hard." "I tried hard to understand the information presented in this lesson." Participants 
responded on a 5-point Likert scale which rated responses from "Strongly Agree" (5) to "Strongly Disagree" (1). In this 
study, the questionnaire had a Chronbach's alpha of .78 , indicating an acceptable degree of internal consistency (Borg 
and Gall, 1983). 

Additional questions on the self-report questionnaire gathered infoimation regarding participants' gender, previous 
experience with Macintosh computers, experience using desktop publishing software, and other demographic information. 

Secondary task mfta.tnre The secondary task measure consisted of a computer program that presented an audible 
beep at random intervals then measured and the time between the beep and the moment when the participant clicked the 
mouse button in "tics" (a tic is equivalent to 1/60 of a second). During the practice phase, participants received seven 
practice beeps randomly general^ in 10 to 30 second intervals. By averaging the seven practice response times, a 
baseline response rate was established. 

During the actual use with instructional materials, participants responded to a beep randomly generated over 
intervals ranging from 30 to 90 seconds. Secondary task reaction times were calculated for each student by averaging the 
response time for each participant engaged with a lesson. Consistent with a procedure used by another researcher 
(Grimes, 1990), responses that were more than three standard deviations from the individual's mean were ignored for 
analysis. For example, if a participant failed to respond to a beep or responded after a much longer interval than usual, 
that particular score was eliminated from the final average score. 

Post-test . The post-test consisted of a cued recall test, where students wrote down all they could remember 
about topics presented in the lessons. Participants were first asked to recall the ideas that impressed them most. The 
next five questions corresponded to the five major topics of the videotape. Participants responded to questions such as; " 
In SuperPaint, what tools do you remember and what were their uses?" and "What steps would you follow to create a 
news letter with a page-layout program?" Finally students were provided with the opportunity to list other information 
they recalled from the lesson ("What else do you remember from the lesson?" ) 

A scoring key was developed that categorized responses to the post-test. The primary researcher and two research 
assistants (a graduate student in instructional design and an undergraduate education student) each reviewed 15 recall post- 
tests and clustered responses that seemed to be of a similar type. Responses seemed to directly recall information 
presented in the lessons, make inferences based on the information presented in the lessons, summarize information that 
was presented in the lessons, or evaluate the lessons. Differences in coding responses among the three individuals were 
discussed and coding categories were finalized. Tlie final coding categories consisted of recall responses (this software 
leaves room to correct mistakes, I can design my own letterhead), inferences (it seemed easy, it seems convenient to use), 
and summary statements (there were different photos of parts of the computer, I remember the letters from people in the 
industry). Evaluative comments (the information was presented in good form; I was falling asleep) were not coded. The 
two research assistants then coded 15 other post-test using the coding scheme. After finding an acceptable inter-rater 
reliability of .97 between the two scorers, the tests were scored by one of the research assistants using the scoring key. 
Participants received a recall score, inference score, summary score, and total post-test score. 
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Procedures 

All sessions were held in a computer lab which contained approximately 40 computers. Groups using the 
secondary task measure had a demonstration of the data collection program ,md practice with the program. During the 
practice session, a baseline measure of reaction time to the secondary task was collected. 

All participants received either a motivationally enhanced videotape, "standard " videotape, motivationally 
enhanced print booklet, or "standard" print booklet. All learners were asked to read or view the lesson to learn as much as 
possible about the topic of the lesson. Participants in four of the eight sessions were asked to respond to a secondary 
task while ahending to their lesson. They were informed that their primary task was to learn as much as possible from 
the lesson and also to press a key on a computer keyboard as soon as they heard a beep. Participants participating in the 
secondary task sessions wore earplugs to avoid sound carrying throughout the room. 

Participants receiving one of the videotaped versions viewed the lesson as a group from a large video monitor 
placed in the front of the room. Participants receiving the text versions read the lesson at their own pace from individual 
booklets. Following completion of one the four versions of the lesson, all participants completed the mental effort 
questionnaire. Finally, all participants completed, the post-test, then created a logo and letterhead using the software 
demonstrated in the lessons. The participants were provided an opportunity to apply the skills demonstrated in the lesson 
to create a logo and letterhead; however, these products were not analyzed. 

Results 

Multivariate Analysis of Variance (MANOVA) was used to determine if there were significant differences among 
the eight groups for post-test and self-:eport mental effort scores. The analysis indicated several significant effects. 

Interaction between media and version for post-test scores . The interaction between media (text or video) and 
lesson version (standard or motivationally enhanced ) was significant, F (2, 107) = 4.37, p=.01. Univariate analysis 
indicated that the interaction effects were caused by significant differences among groups on the post-test scores, F (1, 
108)= 8.38, p=.005. An examination of the means (see Table 1) revealed that post-test scores were higher for the 
standard version of video (M= 30.58) than the motivationally enhanced version (M= 21.23) , and higher for the 
motivationally enhanced version of text (M=29.64) than for the standard version (M= 24.49). (See Figure 1.) 

Figure 1 ; Mean Post-test Scores 




Mental effort a.ssessment condition and recall scores . There were also significant differences in the post-test 
scores between groups who responded to the secondary task measure of effort and those who only responded to the mental 
effort questionnaire , F (2. 107)= 4.10, p=.02. Univariate analysis revealed that the significant difference was due to 
significant differences in achievement test scores, F (1,108) = 6.71, p=.01. Participants who responded to the secondary 
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task while reading or viewing the lesson received significantly lower scores (M= 23.33) on the post-test of cued recall 
than those participants who only responded to the mental effort questionnaire after the lesson (M= 28.70). (See Table 1 .) 
Table 1; Post-test means 





standard 

text 


enhanced 

text 


standard 

video 


enhanced 

video 


TOTAL 


Secondary 


23.94 


27.67 


23.11 


15.75 


23.33 


task 


10.80 


9.53 


8.77 


11.60 


10.66 




17 


12 


9 


8 


46 


Questionnaire 


25.00 


30.76 


34.53 


24.36 


28.70 




13.24 


12.17 


12.98 


13.36 


13.29 




18 


21 


17 


14 


31 


Totals 


24.49 


29.64 


30.58 


21.23 






11.95 


11.23 


12.77 


13.16 






35 


33 


26 


22 





F.ffect of media on mental effort scores. MANOVA indicated a significant effect for media, F(2, 107)= 6.83, 
p=.002. Univariate analysis revealed that the significant effect was due to differences among groups on their responses to 
the mental effort questionnaire, F(l, 108)= 1 1.78, p=.001. Students who received the text version of the lesson reported 
spending significantly more effort (M=29.53) in the lesson than those who received the videotaped version (M=26.23). 
(See Table 2.) 

Table 2: Means on Mental Effort Questionnaire 



Text Video 

Means 29.53 26.23 

SD 5.59 4.52 

N 68 48 



Note: Possible scores ranged from 9 to 45 

Secondary ta<;V data Initially, Analysis of Variance was used to analyze differences among group means for 
baseline reaction times collected during the practice session. There were no significant differences in the baseline reaction 
times among groups for media, F( 1, 42) = 1.43. p=.24, lesson version F(l, 42)= .08. p=.78, or the interaction between 
media and lesson version, F(l, 42)= .31, p=.58. Since there were no significant differences among groups for the 
baseline reaction times (see Table 3 for mean scores on practice task). Analysis of Variance was used to determine 
differences among group means for reaction times to the secondary task. The interaction between media and lesson 
version was significant, F(l,42)= 4.17, p=.047. There also were significant differences among groups for media, F 
(1,42)= 7.28, p=.01, and lesson version, F(l,42)= 1 1.39, p =.002. An examination of the group means revealed that 
reaction times for those who received the video versions were significantly longer (M= 78.13) than those who received 
the text versions (M=55.38). In addition, the reaction time means for those who received the motivationally enhanced 
versions of the lesson (M=78.33) were significantly longer than those who received the standard versions of the lesson 
(M=52.59). (See Table 4 for means for secondary task.) Although the analysis revealed a significant intcra'-tion between 
media and lesson version (see Figure 2), an examination of the means indicates that the effects of media and lesson 
version was much stronger than the interaction. The significant interaction effect appears to be caused by the larger 
difference in reaction time scores between the standard video group (M=56.27 ) and the enhanced video group (M=102.7 1 ) 
than between the standard text group (M= 50.< 5) and the enhanced text group (M= 62.08). 
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Table 3; Mean baseline reaction scores 



Table 



O 

ERIC 



Text Video 



Standard 


43.62 


47.76 


SD 


14.24 


18.05 


n 


17 


9 


Enhanced 


41.84 


53.24 


SD 


27.50 


25.45 


n 


12 


8 



Note: reaction time measured in "tics", equivalent to 1/60 of a second 



4: Mean reaction times to secondary task 






Text 


Video 


Total 


Standard 


50.65 


56.27 


52.59 


sd 


18.68 


23.83 


20.31 


n= 


17 


9 


26 


Enhanc<^d 


62.08 


102.71 78.33 




sd 


34.42 


36.79 


40.02 


n= 


12 


8 


20 


TMals 


55.38 


78.13 






26.41 


38.04 






29 


17 




Note: 


reaction time measured in “tics"i equivalent to 1/60 of a second 
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Figure 2 ; Mean Reaction Times to Secondary Task 




Correlations . Correlations among scores on the three dependent measures (post-test, mental effort questionnaire, 
and secondary task reaction measure) were tested using a Pearson's r test of correlation. The correlation between reaction 
time means on the secondary task measure and the scores on the self-report of mental effort were very low, r=-.05, p=.61 
(N=46). Likewise, the correlations were low between post-test scores and responses to the self-report measure of effort 
r=-.069, p= .46 (n=l 1 7). The correlation between post-test scores and the secondary task response times was somewhat 
higher, r=-.28, p= .056 (n= 46) but in a negative direction. Correlations between the number of inferences generated on 
the post-test and scores on the mental effort questionnaire and secondary task measure were also low, with values of r=- 
.06, p=.79 (questionnaire) and r= .22, p= .52 (secondary task) respectively. 

Post hoc analysis 

Several analysis were conducted to determine whether participants' prior experience with desktop publishing or 
Macintosh computers may have influenced participants scores on the post-test and mental effort questionnaire. When 
prior experience with desktop publishing (experienced/ no experience) and Macintosh computers (experienced/ no 
experience) were used as independent variables in a MANOVA with mental effort and post-test scores as the dependent 
variables, there were no significant interactions, F(2, 103)= 1.20, p=.31, differences due to prior experience wiuh 
Macintosh computers, F(2, 103)= 2.17, p=.12, or desktop publishing software (F(2,103) = .32, p=.73). 

Discussion 

The purpose of this study was to investigate a) whether college-age learners invest more effort in print-based 
lessons desig ned to teach the skills involved in desktop publishing than in video-based versions of the lessons, b)whether 
learners inves' more mental effort in motivationally enhanced lessons than in "standard" lessons which lack motivational 
enhancements, c) the extent to which learners’ self-reports correlate with their "on line" measurements of effort collected 
using a secoiidary task technique and d) the correlation between mental effort scores (from secondary task or self-report 
measures) and post-test scores on a recall measure. 

As in Beentjas and van der Voort's (1993) study, the two measures of mental effort ap eared to assess different 
things. Correlations between the mental effort questionnaire and the secondary task measure of effort were very low. 
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Despite the lack of correlation between scores on the mental effort questionnaire and the secondary task theasurc 
of effort, the findings of this study, conducted with college age students, are consistent with the results of studies 
conducted with elementary and middle school students. Like the younger learners who participated in Salomon (1983; 
1984: Salomon & Leigh. 1984) and Beentjas and van der Voort’s (1993) studies, college students who participated in this 
study repotted that they invested more mental effort in learning from print versions of the lessons than in video-based 
versions on a self-report questionnaire. And like the younger learners who responded to a secondary task measure of 
effort in Beentjas and van der Voort's (1993) study, the college students who received the video-based lesson had 
significantly longer reaction times to the secondary task measure. 

But the question remains: Which is a more valid way of assessing mental effort? If sensitivity to 
manipulations designed to increase the effort expended in the lesson is used as an indictor of the most valid measure of 
mental effort, then the secondary task measure may be superior. Using this means of assessing mental effort, 
participants spent mote mental effort in reading and viewing the motivationally enhanced versions of the lesson than in 
the sta^'ird versions. 

However, the results of this study indicate that the secondary task measure of effort interferes with student 
learning. For students who responded to the secondary task while reading or viewing the lesson, overall achievement was 
lower than for the other groups. Constant response to the secondary task may have interrupted their concentration just 
enough to disrupt the learning process. 

In addition, there appears to be very little relationship between learner’s achievement test scores, as measured by 
a test of cued recall, and the extent to which learners are cognitively engaged in the lesson, as measured by a secondary 
task technique. Likewise, there appeared to be very little relationship between learners' achievement scores and their 
perceptions of the mental effort they expended in processing the lesson as reported on a self-report questionnaire. 

However, these results are consistent with the low correlations between achievement scores and mental effort found in 
Beentjas and van der Voort's (1993) research with younger learners. 

Of course recall depends not only on the extent to which learners concentrate on, process, or elaborate on the 
materials but also on their ability and willingness to retrieve the information (Driscoll, 1994). Participants may have 
lacked motivation to succeed on the achievement test, as course credit was awarded for participation in the study regardless 
of scores on the achievement test 

Keller (1983) also reminds us that achievement scores reflect a variety of factors other than the extent to which 
learners' expend effort in processing a lesson. He (1983) suggests that effort, rather than performance, is an indicator of 
motivation. Although participants who received the motivationally enhanced videotape expe.iued more effort in the 
lesson, as measured by a secondary task technique, than those who received any other version, recall scores were lowest 
for those who received the motivationally enhanced videotape. 

The overall intent in adding "motivational enhancements" was to increase the personal relevance of the materials 
(Nwagbara 1993), theoretically increasing the mental effort expended on the materials and achievement scores. However, 
the addition of motivational aspects may have "backfired". With the addition of motivational enhancements, learners may 
have had a difficult time distinguishing relevant information from irrelevant information. For example, learners who 
received the motivationally enhanced v' Jeotape may have been thinking about how desktop publishing could enhance 
their lives and missed information necessary for success on the recall test. Perhaps increased cognitive engagement, 
especially in response to video, resulted in lower achievement scores. As learners took the time to elaborate on the 
information presented in the motivationally enhanced videotape, they ran the risk of missing critical information due to 
the constant stream of visual and auditory information presented via video. Although learners who received the standard 
version of the videotaped lesson recalled more information than those who received the motivationally enhanced video 
version, learners who received the motivationally enhanced version in print form recalled more information than those 
who received the standard version of the print lesson. In addition, learners who received the motivationally enhanced print 
version had longer reaction times to the secondary task, suggesting greater mental effort, than those who received the 
standard print version. When learning from print, readers may take time to elaborate on the content of a lesson without 
"missing" critical information, due to the individual's control over the pace of their reading. This study provided no 
information on the nature of thoughts that may have occurred during the lesson. Future research using a "think-aloud" 
protocol may provide additional insight as to the cognitive processing that occurs while reading or viewing a lessons. 

At minimum, there is a complex relationship between effort and achievement. In some studies, longer reaction 
times to a secondary task have paralleled less achievement (Lang, et. al., 1993; Reeves, et. al., 1985, and others); 
however, in other situations, increased effort paralleled increased achievement (Grimes, 1990; Britton, Glynn, Muth, & 
Penland, 1985, and others). Other times, more or less effort made no difference in achievement (Britton, Glynn, Meyer. 
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& Penland, 1982; Reeves, et. al., 1985). Does the secondary task technique and achievement tests the simply measure 
different things? The low correlations between learners' post-test scores and their reaction times to the secondary task 
appear to support this idea. 

Future researchers may need to decide which is important to them; Effort or achievement? Does effort matter 
without achievement? Does achievement matter if learners are cognitively engaged and expending effort in learning? It 
may not be useful to use the secondary task technique to determine if manipulations increase the amount of effort 
invested in learning from materials in situations where the major goal is achievement. 

Additional problems in using the secondary task measure of effort remain. Large standard deviations in scores 
are common when using a secondary task measure of effort. Due to the small nu.nber of students in this study, and the 
large standard deviations, the results of this study should be replicated with a larger number of students to increase 
confidence in the findings. In this study, it was possible to hold the class in a computer lab to avoid the appearance of a 
laboratory setting because the subject matter of the lessons was computer skills; however, researchers must consider that 
the use of the secondary task measure requires a laboratory like setting, where participants have access to the data 
collection equipment. In addition, there is a great deal of variability in the way researchers score and analyze data from 
tlie secondary task. If investigations of ways to increase the effort invested in video-based materials are to continue, the 
search for techniques to accurately assess the effort expended in the lessons continues to be an important area for further 
study. 
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